Graphical user interfaces including touchpad driving interfaces for telemedicine devices

ABSTRACT

The present disclosure describes various aspects of remote presence interfaces (RPIs) for use on portable electronic devices (PEDs) to interface with remote presence devices. An RPI may allow a user to interact with a telepresence device, view a live video feed, provide navigational instructions, and/or otherwise interact with the telepresence device. The RPI may allow a user to manually, semi-autonomously, or autonomously control the movement of the telepresence device. One or more panels associated with a video feed, patient data, calendars, date, time, telemetry data, PED data, telepresence device data, healthcare facility information, healthcare practitioner information, menu tabs, settings controls, and/or other features may be utilized via the RPI.

RELATED APPLICATIONS

This U.S. patent application is a continuation of U.S. patentapplication Ser. No. 16/045,608, filed Jul. 25, 2018, titled “GraphicalUser Interfaces Including Touchpad Driving Interfaces for TelemedicineDevices,” which is a continuation of U.S. patent application Ser. No.15/154,518, filed May 13, 2016, titled “Graphical User InterfacesIncluding Touchpad Driving Interfaces for Telemedicine Devices,” whichis a continuation of U.S. patent application Ser. No. 14/550,750 filedNov. 21, 2014, titled “Graphical User Interfaces Including TouchpadDriving Interfaces For Telemedicine Devices,” which is a continuation ofPCT Application No. PCT/US2013/031743 (the “PCT Application”), andclaims priority under 35 U.S.C. § 119(e) to: U.S. ProvisionalApplication No. 61/650,205 filed May 22, 2012, titled “Remote PresenceInterface and Patient Data Integration;” U.S. Provisional ApplicationNo. 61/674,794 filed Jul. 23, 2012, titled “Graphical User InterfacesIncluding Touchpad Driving Interfaces for Telemedicine Devices;” U.S.Provisional Application No. 61/674,796 filed Jul. 23, 2012, titled“Clinical Workflows Utilizing Autonomous and Semi-AutonomousTelemedicine Devices;” U.S. Provisional Application No. 61/674,782 filedJul. 23, 2012, titled “Behavioral Rules For a Telemedicine Robot ToComply With Social Protocols;” U.S. Provisional Application No.61/766,623 filed Feb. 19, 2013, titled “Graphical User InterfacesIncluding Touchpad Driving Interfaces for Telemedicine Devices.” All ofthe foregoing applications are hereby incorporated by reference in theirentireties.

TECHNICAL FIELD

This disclosure relates to interactive and display interfaces fortelepresence devices in healthcare networks. More specifically, thisdisclosure provides various graphical user interfaces and interactiveinterfaces for remote presence devices.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the disclosure aredescribed herein, including various embodiments of the disclosureillustrated in the figures listed below.

FIG. 1 illustrates an embodiment of a home page of a portable electronicdevice (PED), including an application providing a remote presenceinterface (RPI) for interacting with a telepresence device.

FIG. 2A illustrates an embodiment of an initial launch page for an RPIor other application associated with a telepresence device.

FIG. 2B illustrates an embodiment of an initial launch page for an RPIor other application associated with a telepresence device intended toinduce a user to orient a PED in a portrait orientation.

FIG. 2C illustrates a front-facing camera located near the top of a PEDin a portrait orientation capturing an image of a healthcarepractitioner.

FIGS. 3A and 3B illustrate embodiments of a login page for the RPI.

FIG. 4 illustrates an embodiment of an endpoint list of varioustelepresence devices and their connectivity state.

FIG. 5 illustrates a connection wizard configured to facilitate theconnection of a user to a telepresence network.

FIG. 6 illustrates an exemplary embodiment of a module within the RPIconfigured to provide visual and interactive control of a telepresencedevice.

FIG. 7 illustrates an embodiment of a map and navigation module of theRPI.

FIG. 8 illustrates an example of a notification that may be displayed toa user via the RPI.

FIG. 9 illustrates an embodiment of a media manager module of the RPIconfigured to allow a user to capture and manage audiovisual media fromthe telepresence device.

FIG. 10 illustrates an embodiment of menu of the RPI allowing a user tomodify various settings on the local PED and/or the remote presencedevice.

FIG. 11 illustrates an embodiment of the RPI in full-screen video mode.

FIG. 12 illustrates an embodiment of the RPI in full-screen data mode.

FIG. 13 illustrates an embodiment of the RPI including a visualrepresentation of patient telemetry data.

FIGS. 14A-C illustrate exemplary embodiments of a telepresence device.

FIG. 15 illustrates an RPI configured to provide patient visualization,remote settings control, navigation control, and access to patient dataintegration.

FIG. 16 illustrates a navigation module within the RPI configured toallow a user to navigate a telepresence device using one or morenavigational techniques.

FIG. 17 illustrates the RPI displaying a video of a patient, associatedtelemetry data, and associated records.

FIGS. 18A and 18B illustrate embodiments of what may be displayed duringa consult on a telepresence device and via an RPI on a PED,respectively.

FIG. 19 illustrates an embodiment of a telepresence device in ascreensaver mode.

FIG. 20 illustrates various embodiments of avatars and/or personalitiesthat may be assigned or used by a medical practitioner and/ortelepresence device.

FIG. 21 illustrates an RPI configured to utilize various pointer-basednavigational modes for navigating a telepresence device.

FIG. 22 illustrates an RPI configured to utilize destination-basednavigational modes for navigating a telepresence device.

FIG. 23 illustrates a selectable destination patient list that can beused within an RPI to navigate a telepresence device.

FIG. 24 illustrates an RPI configured to utilize map-based navigationalmodes for navigating a telepresence device.

FIG. 25 illustrates a full-screen view of a hallway from a telepresencedevice as visualized on a PED via an RPI.

FIG. 26 illustrates a full-screen view of various doorways in ahealthcare facility, as visualized on a PED via an RPI.

FIG. 27A illustrates an intended or directed navigation path of atelepresence device as visualized on a PED via an RPI.

FIG. 27B illustrates a mouse-based driving interface for navigating atelepresence device via an RPI.

FIG. 27C illustrates the mouse-based driving interface with long drivevector drawn on the video feed to indicate a direction and a relativelyfast velocity.

FIG. 27D illustrates the mouse-based driving interface with a shortdrive vector drawn on the video feed to indicate a direction and arelatively slow velocity.

FIG. 27E illustrates the mouse-based driving interface with a vector forrounding a corner within the RPI.

FIG. 27F illustrates the mouse-based driving interface with a vectordrawn to cause the telepresence device to spin in place.

FIG. 27G illustrates the mouse-based driving interface with a vectordrawn towards an object.

FIG. 27H illustrates the mouse-based driving interface used to reversethe telepresence device with a camera of the telepresence deviceoriented in reverse and slightly downward.

FIG. 28 illustrates a bedside view of a patient bed and an associatedpatient monitor, as visualized on a PED via an RPI.

FIG. 29 illustrates a click-to-zoom feature of an RPI that can be usedwhen medical practitioners and/or other users visualize a patient on aPED via an RPI.

FIG. 30 illustrates a StrokeRESPOND application for accessing atelepresence device that may be a separate application or integratedwithin an RPI.

FIG. 31 illustrates a transparent user image overlaid on an imagegenerated by a telepresence device.

FIG. 32 illustrates a toolbar for managing modules and controloperations available via an RPI, while simultaneously displaying a videofeed from a telepresence device.

FIG. 33 illustrates a toolbar associated with a media management moduleof an RPI.

FIG. 34 illustrates a toolbar separating an endpoint list and a patientlist in an RPI, allowing for quick user selection of various possiblepermutations.

FIG. 35 illustrates a view of a touch pad control pane for navigating atelepresence device while displaying a video feed from a telepresencedevice in an upper window.

FIG. 36 illustrates an avatar display of telepresence devices in a lowerwindow and a video feed from a telepresence device in an upper window.

FIG. 37 illustrates a visualization of a telepresence device overlaid ona video feed from a telepresence device that may be useful fornavigation of the telepresence device.

FIG. 38 illustrates a toolbar of an RPI associated with a settingsmanager for managing settings on a local PED and/or a telepresencedevice.

FIG. 39 illustrates a navigational mode including a landing stripnavigational panel allowing a user to specify a direction of atelepresence device.

FIG. 40 illustrates a full-screen video feed from a telepresence device,including an overlaid toolbar.

FIG. 41 illustrates a joystick-style control on a touch interface of aPED for controlling a telepresence device via an RPI.

FIG. 42 illustrates a dual joystick-style control on a touch interfaceof a PED for controlling a telepresence device via an RPI.

FIG. 43 illustrates a state diagram for an RPI for use on a PED.

FIG. 44 illustrates a full-screen video feed from a telepresence device,including an overlaid toolbar and joystick control.

FIG. 45 illustrates an exemplary toolbar of icons that may be overlaidwithin an RPI.

FIG. 46 illustrates an overlaid instructional panel associated withdriving a telepresence device via an RPI on a PED.

FIG. 47 illustrates a multi-participant telepresence session conductedvia an RPI on a PED.

FIG. 48 illustrates a window accessible via an RPI on a PED providingaccess to a care team of a particular patient.

FIG. 49 illustrates an exemplary overlay help screen accessible withinthe RPI on a PED to provide instructions regarding available functionson any given screen.

The described features, structures, and/or characteristics of thesystems and methods described herein may be combined in any suitablemanner in one or more alternative embodiments, and may differ from theillustrated embodiments.

DETAILED DESCRIPTION

Healthcare facilities may include telemedicine technologies, such astelepresence devices in a telepresence network, that allow remotehealthcare practitioners to provide services to patients and/or otherhealthcare practitioners in remote locations. For example, a remotehealthcare practitioner may be a neurologist practicing in a relativelylarge hospital who may, via a telepresence device, provide services andconsultations to patients and/or other medical professionals in arelatively small hospital that may otherwise not have a neurologist onstaff.

The present disclosure provides various interfaces for visualizing,controlling, and driving telepresence devices. In a remote presence (RP)system, a telepresence device, such as an autonomous or semi-autonomousrobot, may communicate with an interfacing device via a communicationsnetwork. In various embodiments, a portable electronic device (PED) maybe used to run a remote presence interface (RPI) adapted to providevarious telepresence functions.

In various embodiments, an RPI application may be executed on a deviceconfigured as a stand-alone PED configured to solely run the RPIapplication. Alternatively, the RPI application may be executed by anyof a wide variety of PEDs configured as multi-purpose PEDs, such as theApple iPad®. In various embodiments, a user may launch an RPIapplication and login using login credentials. The login process mayutilize any of a wide variety of encryption algorithms and dataprotection systems. In various embodiments, the login process may meetor exceed standards specified for Health Insurance Portability andAccountability Act (HIPAA) compliance.

A user may select one or more endpoint telepresence devices via the RPI.Once a secure connection is established, the RPI may display a videofeed from the telepresence device on the PED. In addition, the RPI mayinclude any number of navigational panels, setting controls, telemetrydata displays, map views, and/or patient information displays.

In some embodiments, the RPI may allow a user to manually,semi-autonomously, or autonomously control the movement of thetelepresence device. The RPI may allow a user to specify movement (i.e.,a location within a healthcare facility or a physical movement, such asa head turn, of the telepresence device) using a destination selectionpanel, an arrow, a physical or virtual joystick, a touch pad,click-to-destination, vector based driving, mouse-based vector driving,and/or other navigational control.

For example, a user may provide a navigation input by selecting alocation within a live video feed (e.g., via a click or a touch). Thenavigation input may be used to transmit navigation instructions to aremote presence device. For instance, in a click drive mode, theselection of a location within the live video feed may be used tonavigate the telepresence device to the selected location. The selectionof a location on a floor or hallway may result in navigationinstructions that cause the telepresence robot to autonomously orsemi-autonomously navigate to the selected location.

In some embodiments, an operator may input a navigation input in theform a navigation vector provided (e.g., drawn, traced, mapped) withrespect to the live video feed. The navigation vector may include alength on the display and/or other input device (e.g., touchpad) and anangle with respect to plane of the display and/or other input device. Asan example, a plane of the display and/or other input device may bedescribed with a Cartesian coordinate system as having an x-axis and ay-axis. The navigation vector may be provided with respect to thedisplay plane and described as having an x-axis component (horizontaldirection) and a y-axis component (vertical direction).

According to various embodiments, a navigation input provided as anavigation vector may be decomposed into a horizontal (x-axis) componentand a vertical (y-axis) component. The length and sign (i.e., positiveor negative value) of the horizontal component of the navigation vectormay be used to determine a magnitude and direction of a rotationalvelocity and/or an angular displacement of a telepresence device. Thelength of the vertical component of the navigation vector may be used todetermine the magnitude of a forward velocity and/or a forwarddisplacement of a telepresence device.

The length of the horizontal component may be used to determine themagnitude of the rotational velocity and/or angular displacement using ascaling function. The scaling function may be constant, linear, and/ornon-linear. Thus, using a non-linear scaling function, a firsthorizontal component twice as long as a second horizontal component maynot result in a first rotational velocity double that of the secondrotational velocity. Similarly, the length of the vertical component maybe used to determine the magnitude of the forward velocity and/orforward displacement using a scaling function. Again, the scalingfunction may be constant, linear, and/or non-linear. The scalingfunction used to translate the horizontal component may be differentthan the scaling function used to translate the vertical component.

Alternatively, selectively, and/or in a different navigation mode, theselection of a location within the live video feed may be used togenerate a navigation vector, where the length of the vector correspondsto the velocity at which the telepresence device should navigate and/orthe distance the telepresence device should navigate, and the angle ofthe vector corresponds to the direction the telepresence device shouldnavigate. For instance, a navigation input may be in the form of avector drawn as either a final endpoint selection or as a vectorincluding a beginning point and an end point. The end point of thevector may represent the location to which the telepresence deviceshould navigate (i.e., a desired navigation point). Navigationinstructions may be generated based on the current location of thetelepresence device and the endpoint of the vector within the live videofeed. According to some embodiments, the vector's image pixel endpointmay be mapped via a lookup table to one or more translate and rotatecommands to cause the telepresence device to navigate to the selectedlocation. In some embodiments, the length of the vector may correspondto the desired distance to move the telepresence device, a desiredacceleration of the telepresence device, and/or a desired velocity ofthe telepresence device.

As an example, a navigation input may be received that directs atelepresence device to navigate forward (with respect to the live videofeed) 3 meters and to the right 2 meters. Any of a wide variety ofnavigation instructions may be possible to correctly navigate thetelepresence device. For instance, the navigation instructions maydirect the telepresence device to navigate approximately 3.6 meters at a34 degree angle relative to the video feed. Alternatively, thenavigation instructions may direct the telepresence device to navigate 3meters forward and then 2 meters to the right. As will be appreciated,any number of navigations routes and corresponding navigationinstructions may be possible. The navigation input may be translatedinto navigation instructions as a plurality of drive forward commandscoupled with rotate commands.

In various embodiments, the navigation instructions are derived from anavigation input indicating a desired direction and/or location and thecurrent location of the telepresence device. In some embodiments, thelive video feed may be delayed slightly due to network and/or processinglimitations. For example, a live video feed may be delayed by a fewtenths of a second or even by a few seconds. This video latency mayresult in inaccurate navigation instructions. Accordingly, thenavigation input may be adjusted based on the known video latency andthe velocity and direction of the robot.

Returning to the example above, if the telepresence device weretraveling at 1.25 meters per second in a forward direction relative tothe live video feed and the video latency was 0.8 seconds, then thetelepresence device will have already traveled 1 of the desired 3 meterswhen the selection is made. Accordingly, the selection of a location 3meters ahead and 2 meters to the right, may be translated or mapped tonavigation instructions that cause the telepresence device to travel 2meters forward and 2 meters to the right (or 2.8 meters at a 45 degreeangle) to compensate the movement of the telepresence device. Thus, thenavigation instructions may be adjusted based on the latency of thevideo feed.

In various embodiments, a navigation input in the form of a vector (suchas a vector provided in a mouse-based vector input mode) may betranslated into navigation instructions through the use of a lookuptable. In some embodiments, the lookup table may include values thatcompensate for latency. That is, the navigation instructions returned bythe lookup table may be based on the navigation input and the current orrecent average video latency.

Various alternative navigation systems, methods, and processing stepsmay be used in conjunction with the presently described remote presenceinterface, including those described in U.S. Pat. No. 6,845,297, titled“Method and System for Remote Control of Mobile Robot,” filed on Jan. 9,2003, and European Patent No. 1279081, titled “Method and System forRemote Control of Mobile Robot,” filed on May 1, 2001, whichapplications are hereby incorporated by reference in their entireties.

The RPI may provide various notifications associated with the networkconnection, the PED, a patient, a healthcare facility, a healthcarepractitioner, a telepresence device, and/or the like. The RPI mayinclude a media management module configured to allow a user to recordand/or store audio and/or visual data for subsequent use. A settingspanel may allow settings on the PED and/or the telepresence device to beadjusted. In some views, multiple windows may provide quick access tovarious panels of information. For example, one or more panelsassociated with a video feed, patient data, calendars, date, time,telemetry data, PED data, telepresence device data, healthcare facilityinformation, healthcare practitioner information, menu tabs, settingscontrols, and/or other features may be displayed simultaneously and/orindividually in a full-screen mode.

The RPI may utilize a camera of the PED to capture an image of the userof the PED and project the image on a screen on the telepresence device.In some embodiments, the image on the screen of the telepresence devicemay be modified and/or enhanced. In one embodiment, an avatarrepresenting the user of the PED is displayed on the PED. In someembodiments, the RPI may encourage or induce a user to utilize the PEDwith a front-facing camera at or above eye-level. Similarly, the RPI mayencourage or induce a user to utilize the PED with a front-facing cameraapproximately centered on a user's face. For instance, on an AppleiPad®, the RPI may encourage a user to utilize the iPad® in a portraitmode for many tasks in order to maintain a more natural perspective ofthe user for projection via the screen on the telepresence device.

In various embodiments, the RPI may facilitate conference sessions withmore than two people interacting via a combination of PEDs and/ortelepresence devices. For example, multiple healthcare practitioners mayparticipate in a remote consultation. In such an example, eachhealthcare practitioner may utilize an RPI on a PED to access atelepresence device at a bedside of a patient. The remote presencesystem, via the network, servers, RPIs, and/or telepresence devices, mayfacilitate the multi-user experience.

The RPI may incorporate sub-applications and/or provide access torelated applications, such as a StrokeRESPOND application configured toprovide one or more functions and/or workflow processes described inU.S. patent application Ser. No. 12/362,454, titled “DOCUMENTATIONTHROUGH A REMOTE PRESENCE ROBOT,” filed on Jan. 29, 2009, whichapplication is hereby incorporated by reference in its entirety.

As described herein and illustrated in various embodiments, the displayof a PED may be utilized by the RPI to display any combination of videofeed panels, informational panels, data panels, setting control panels,navigation panels, and/or panels providing access to any of variousfunctions made accessible via the RPI. The RPI may be configured tomaintain a “stateful” connection at the application layer, such that asession and/or variables may be continued and/or maintained in the eventthat the connection is lost or dropped. The RPI application may attemptto re-establish a disconnected session using saved or logged variablesin the event a connection is lost or dropped. The PED and/or RPIapplication may have settings that enable a user to maximize frame rateor image quality at the expense of the battery life of the device, orvice versa.

The RPI may include an information bar (“status bar”) that displaysvarious status information related to the PED, including battery life,wireless connection strength, wireless connection name or SSID, or thecurrent time. The RPI may include one or more toolbars. A toolbar may bedisposed along the top edge of the upper pane of the RPI. The toolbarmay be manually hidden by touching or selecting an icon or a specifiedarea of the display, or the toolbar may auto-hide after a period oftime. Once hidden, the user may un-hide the toolbar by touching,selecting, or otherwise moving an input device on or around an icon orspecified area of the display.

The RPI may include a “Picture-in-Picture” region or window thatdisplays local video or image data currently being captured by thecamera of the PED. The local video feed may be captured from a cameraeither incorporated within or otherwise in communication with the PED.The user may resize the local video window, reposition it within thedisplay, and/or remove it. The local video window may be displayed in alower pane of the GUI, while the remote video is displayed in the upperpane, or vice versa.

The RPI may include an in-video interface that enables the user tocontrol the endpoint device by interacting with the live video feed. Forexample, when the user touches, taps, clicks, or otherwise selects apoint in the live video feed, the endpoint may change a mechanical panor tilt, or digital pan or tilt, or both, such that the point selectedby the user is centered in the live video feed. The user may also adjustan optical or digital zoom, or both, of the live video feed. Forexample, a user may adjust an optical and/or digital zoom by pinchingtogether or spreading apart two or more fingers on the surface of atouch sensitive display of the PED. As another example, the user maycontrol all or some of a mechanical or digital pan, tilt, or zoom of thelive video feed by pressing and dragging a finger on the surface of thedisplay to specify a diagonal or other dimension of a zoom region. Aftera period of time or upon releasing his or her finger, the endpoint maymechanically or digitally pan, tilt, or zoom to match a dimension of thezoom region to a dimension of the live video feed, and/or to match acenter of the zoom region to a center of the live video feed. In oneembodiment, a user may zoom out to a default zoom (which may be fullyzoomed out) by performing a double-tap in the live video feed, or bydouble-clicking or right-clicking a mouse cursor within the live videofeed.

In one embodiment, the user may direct movement of a telemedicine device(or other telepresence device) to a particular location within the livevideo feed by performing a “touch-hold-tap,” where the user touches alocation on the screen, holds his or her finger on that location for abrief interval until a cursor appears on the screen under the user'sfinger, positions the cursor (which now follows the user's finger) at apoint in the remote video window representing the desired destination,and subsequently taps his or her finger once again to confirm thelocation as the desired destination. The telepresence device may thenproceed to a position in the live video feed corresponding to thelocation selected by the user.

Additional functionalities available via the RPI through a touch screeninterface may include a two-finger swipe to display video from one ormore auxiliary video sources, such as a video camera, still camera,endoscope, ultrasound device, radiological imaging device, magneticresonance imaging device, or other medical imaging device.

A toolbar may be shrunken and/or expanded (vertically or horizontally,or both), or hidden and unhidden, by touching or tapping an icondisposed at the top of the screen. For example, an icon may have theappearance of an arrow or triangle that points “up” when the toolbar isfully expanded or unhidden, and may point “down” when the toolbar isshrunken or hidden. The icon itself may shrink or expand with thetoolbar. Alternatively, the toolbar may be unhidden by the user “pullingdown” or making a downward swipe from the top (or other edge) of thescreen. The toolbar may again be hidden by “pushing up” or making aswipe toward the top (or other edge) of the screen.

Additional functions or applications may be accessed from the toolbar.Each function or application may have a distinctive icon in the toolbarindicative of the underlying functionality. If there are morefunctions/icons available than can be displayed on the toolbar, thetoolbar may be configured to scroll icons onto and off of the toolbarwith a swipe of the user's finger in the toolbar. The icons may scrollcontinuously in a carousel fashion, or the scrolling may be disabled ina particular direction when there are no further icons to be displayed,thus informing the user of this fact. Alternatively, the toolbar may notscroll available icons, but instead show only a specified set of icons.In this case, additional functions or applications would be exposed tothe user via additional menu levels, windows, or pop-overs, accessiblevia one or more icons contained in the toolbar or elsewhere in the RPI.

The toolbar may provide access to various functions of the RPI such asactivating or deactivating a headset or handset (which may be coupled tothe telepresence device, in either a wired or wireless fashion) whenadditional privacy is desired in communicating with the remote user(i.e., the user at the control station/PED); activating or deactivatinga stethoscope for monitoring a patient; creating a still image orsnapshot from a camera (still or video) on or otherwise coupled to thetelepresence device or any auxiliary input; starting or stoppingrecording video from a camera on or coupled to the telepresence deviceor any auxiliary input; navigation features; opening or bringing up amap or list of destinations (e.g., people's names associated with adevice or location); reversing local camera view (toggle local video tocome from front of the PED or back of the PED); activating ordeactivating remote pointer (cursor on remote display follows or mirrorscursor position on local display, which is positioned by the usertapping or touching the desired location on the display); activating ordeactivating a laser pointer on the remote device (a laser pointer maybe positioned on the telepresence device so as to correspond to aposition of tap or touch within the live video feed); muting orun-muting a microphone of the PED; opening the settings panel;disconnecting; and/or other features, options, and/or settings.

From the lower pane of the RPI, the user may initiate a vector drivemode for controlling a telepresence device. For example, a mouse click,touch-drag motion, or other action may be used to “draw” or otherwiseinput a vector for controlling the motion of a telepresence device. Forexample, a user may touch two fingers in the lower pane and drag them adesired distance and direction on the screen to send a drive commandwith a respective velocity and heading to the mobile telepresencedevice.

The interface may further include a mechanism for deactivating anobstacle detection/obstacle avoidance (ODOA) system, such that the usermay operate in a “nudge mode” and continue to slowly move thetelepresence device despite the telepresence being in contact with orclose proximity to another object. This mechanism may automaticallytime-out such that the ODOA system is re-engaged after a period ofinactivity. Placing the telepresence device in nudge mode may cause acamera of the telepresence device to be automatically positioned to lookdown around the body of the telepresence device and either in thedirection of drive command issued by the user or in the direction of anobject closest to or in contact with the telepresence device (so thatthe user may see what obstacle he or she is commanding the telepresencedevice to nudge). Alternatively, the ODOA system may be entirelydeactivated. In one embodiment, the ODOA system may be deactivated solong as the telepresence device is driven or moved below a specifiedvelocity.

The RPI may be configured for deployment on any of a wide variety ofPEDs, including the Apple iPad®, iPod®, and iPhone®. The RPI may beconfigured for deployment on any of a wide variety of PEDs, such asmobile phones, computers, laptops, tablets, and/or any other mobile orstationary computing device. In some embodiments, the RPI may bepresented via a plug-in or in-browser application within a standard webbrowser.

In some embodiments, the RPI may allow for varying feature setsdepending on the type of PED utilized. For example, on a largertablet-sized PED, the RPI may include any combination of the numerousfeatures described herein. While on a smaller PED, such as an AppleiPhone®, the available features may be limited to suit a particularcontext or use-case. In one embodiment, the RPI may allow a user, suchas nurse or hospital staffer, to control the movement of a telepresencedevice without establishing a telepresence audio/video session. In otherembodiments, for example, a remote family member of a patient mayconduct a two-way voice and video telepresence with his or her iPhone,but may not have permission to drive or otherwise control the movementof the telepresence device. Any number of features or combination offeatures may be included and/or excluded for a particular PED.

As an example scenario, a nurse may utilize an RPI on a PED with limitedfunctionality, such as an Apple iPhone®, to request a cardiac consultfor a patient. A telepresence system in contact with the RPI submits therequest to a telepresence device. The telepresence device may beginnavigating to the patient while simultaneously initiating a connectionwith a cardiac doctor. For example, the telepresence device may call anappropriate cardiac doctor (e.g., the cardiologist on call, the nearestcardiologist, the patient's specific cardiologist, etc.) on one or morePEDs belonging to the doctor. Given that the nurse is using a PED withlimited functionality, the nurse may be allowed to control thetelepresence device and/or participate in an audio-only telepresencesession, but may be provided a limited feature set. Accordingly, thenurse may be able to communicate with the doctor as the telepresencedevice navigates to the patient.

In some embodiments, a nurse or a patient may request a doctor (or aspecific type of doctor) via an RPI or via a display interface directlyon a telepresence device. The RPI, the telepresence device, and/or acorresponding telepresence system may intelligently call a specificdoctor as described herein. Alternatively, the RPI, the telepresencedevice, and/or a corresponding telepresence system may call a pluralityof doctors. In such a scenario, the first doctor to attend may beconnected via an RPI to a telepresence device to service the request ofthe nurse or patient. The call routing and telepresence session may bemanaged in a network cloud, a telepresence network, and/or via someother suitable computing device.

Throughout the specification, various functions, features, processing,and/or other computing actions are described as being performed by atleast one of an RPI, a PED, a telepresence system, and/or othercomputing device. It will be appreciated that in many instances, theactual processing, calling, function implementation, recording, and/orother computer-performed actions may be executed on a local device, aremote device, and/or via networked or cloud device.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, the appearances of the phrases “in oneembodiment” and “in an embodiment” in various places throughout thisspecification are not necessarily all referring to the same embodiment.In particular, an “embodiment” may be a system, an article ofmanufacture (such as a computer-readable storage medium), a method,and/or a product of a process.

The phrases “connected to” and “in communication with” refer to any formof interaction between two or more entities, including mechanical,electrical, magnetic, and electromagnetic interaction. Two componentsmay be connected to each other even though they are not in directcontact with each other and even though there may be intermediarydevices between the two components.

Types of telepresence devices include, but are not limited to, remotepresence devices, mobile telepresence units, and/or control stations.For example, a remote presence device may include a telepresence robotconfigured to move within a medical facility and provide a means for aremote practitioner to perform remote consultations. Additionally,telepresence devices may comprise any of a wide variety of endpointdevices, such as those described in U.S. patent application Ser. No.13/360,579 filed on Jan. 27, 2012, titled “INTERFACING WITH A MOBILETELEPRESENCE ROBOT,” which application is hereby incorporated byreference in its entirety. Telepresence devices may also comprise any ofthe endpoint devices described in U.S. patent application Ser. No.13/360,590 filed on Jan. 27, 2012, titled “INTERFACING WITH A MOBILETELEPRESENCE ROBOT,” which application is hereby incorporated byreference in its entirety.

A “portable electronic device” (PED) as used throughout thespecification may include any of a wide variety of electronic devices.Specifically contemplated and illustrated are tablet-style electronicdevices, including, but not limited to, electronic readers, tabletcomputers, tablet PCs, cellular phones, interactive displays, videodisplays, touch screens, touch computers, and the like. Examples of PEDsinclude the Apple iPad®, iPod®, and iPhone®, the Hewlett Packard Slate®,the Blackberry Playbook®, the Acer Iconia Tab®, the Samsung Galaxy®, theLG Optimus G-Slate®, the Motorola Xoom,® the HP touchpad Topaz®, theDell Streak®, and the like.

Throughout this description, a tablet-style touch-screen PED is used asan exemplary PED; however, any of a wide variety of PEDs and/or otherelectronic devices may be used instead. For instance, tablet computingdevices, cellular phones, computers, laptops, etc., could be used inplace of the illustrated and described touch-screen tablet devices. Itwill be appreciated by one of skill in the art that operations andfunctions performed on or by a PED may also be performed on a stationaryportable electronic device, such as a desktop computer or server.

The embodiments of the disclosure may be understood by reference to thedrawings, wherein like elements are designated by like numeralsthroughout. In the following description, numerous specific details areprovided for a thorough understanding of the embodiments describedherein. However, those of skill in the art will recognize that one ormore of the specific details may be omitted, or other methods,components, or materials may be used. In some cases, operations and/orcomponents are not shown or described in detail.

Furthermore, the described features, operations, or characteristics maybe combined in any suitable manner in one or more embodiments. The orderof the steps or actions of the methods described in connection with theembodiments disclosed may be varied. Thus, any order in the drawings orDetailed Description is for illustrative purposes only and is not meantto imply a required order, unless otherwise specified.

Embodiments may include various features, which may be embodied inmachine-executable instructions executed by a general-purpose orspecial-purpose computer (or other electronic device). Alternatively,the features may be performed by hardware components that includespecific logic for performing the steps or by a combination of hardware,software, and/or firmware. Accordingly, the various components, modules,systems, and/or features described herein may be embodied as moduleswithin a system. Such a system may be implemented in software, firmware,hardware, and/or physical infrastructure.

Embodiments may also be provided as a computer program product includinga non-transitory machine-readable medium having stored thereoninstructions that may be used to program or be executed on a computer(or other electronic device, such as a PED) to perform processesdescribed herein. The machine-readable medium may include, but is notlimited to, hard drives, floppy diskettes, optical disks, CD-ROMs,DVD-ROMs, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards,solid-state memory devices, or other types of media/machine-readablemedia suitable for storing electronic instructions.

FIG. 1 illustrates an embodiment 100 of a home page of a portableelectronic device (PED) 105. The PED 105 may include one or morephysical buttons, such as button 120, and a display screen. The displayscreen may be a touch-screen configured to allow a user to provide inputvia the touch screen. The PED 105 may be configured to display variousicons 115 to launch corresponding applications. In various embodiments,a remote presence interface (RPI) icon 110 may be used to launch an RPIapplication for interfacing with a telepresence device. An RPI accordingto any of the various embodiments described herein may alternatively beutilized on any of a wide variety of computing platforms and using anyof a wide variety of programming tools and languages.

FIG. 2A illustrates an embodiment 200 of an initial launch page 210 foran RPI or other application associated with a telepresence device. Asillustrated, a PED 205 may display the initial launch page 210 until theRPI fully loads. Alternative logos, graphics, informational displaysand/or other content may be displayed. For example, a general greetingor specific greeting may be displayed on/during launch. In someembodiments an icon or object may indicate the progress or timeframeuntil loading is complete. In some embodiments, a cancel button orsettings configuration may be available within the initial launch page210.

FIG. 2B illustrates embodiment 250 of an alternative initial launch page270 including an image of a telepresence device 275. The launch page 270may be oriented such that it induces or encourages a user to align thePED 255 such that camera 280 is oriented in about the middle of a user'sface and at or above an eye level of a user's face. In the illustratedembodiment, if a user had been holding the PED 255 in a landscapeorientation, the image of the telepresence device 275 in the portraitorientation may encourage the user to rotate the PED 255 to the portraitorientation with the camera 280 in the top middle. An information bar260 may provide information such as battery power, time, and/orconnection strength. The initial launch page 270 may be unique to eachPED on which it is utilized. For example, the RPI may detect that a PEDwith a camera positioned for a landscape orientation is being used andreorient the displayed launch page 270 accordingly.

FIG. 2C illustrates a front-facing camera 280 of the PED 200. The camera280 is illustrated as located on the top of the PED 200 for capturing ancapturing an image of a healthcare practitioner just above eye level ina portrait orientation. This location of the camera 280 relative to theface of a healthcare practitioner 290 may facilitate the capture of anaesthetically pleasing image of the healthcare practitioner 290 fordisplay on a screen of a telepresence device. Accordingly, a patient orother person viewing the screen on the telepresence device may view anatural image of the healthcare practitioner 290 using the RPI on thePED 200, rather than an un-aesthetically pleasing perspective of thehealthcare practitioner 290. For example, a side-captured orbottom-captured image of a healthcare practitioner 290 may not beaesthetically pleasing.

According to various embodiments, the RPI may automatically adjust thefield of view (FOV) of the camera 280. In some embodiments, thehealthcare practitioner 290, or other user, may manually adjust thefield of view of the camera 280. Additionally, any number ofcompositional and exposure variables of the camera may be automaticallyor manually adjusted/adjustable. The RPI or the healthcare practitioner290 may automatically or manually select which camera to use in theevent a PED has multiple cameras. A user may, in some embodiments,select which camera is used during a session.

FIGS. 3A and 3B illustrate embodiments of a login page 325 on a PED 305for accessing a telepresence device or telepresence network via an RPI,respectively. Any of a wide variety of login credentials 310 and 315and/or security technologies may be employed. In some embodiments, ausername, handle, and/or password may be provided on a differentsettings page. In some embodiments, the username, handle, and/orpassword may be maintained in a configuration file. The login page mayinclude a button 330 or link that opens a settings or configurationpage. Accordingly, a “remember me” 320 option may be provided to a user.

FIG. 4 illustrates an embodiment 400 of an endpoint list 410 generatedby the RPI running on the PED 405. The endpoint list 410 may includevarious telepresence devices 420 and their respective connectivitystates 415. A user may indicate (via a touch, a click, using a stylus,and/or by speaking) to which of the available endpoints he or she wouldlike to connect. Where an ADT (Admissions, Discharges, and Transfers)data source is available, patients may also be listed. Selecting aparticular patient may initiate a connection to an endpoint device(telemedicine device) that is assigned to, associated with, and/or inproximity to the selected patient. A telemedicine device in proximity tothe patient could be a telemedicine device nearest a location of theselected patient, the same bed, room, or floor number. Where multipleautonomous or semi-autonomous endpoint devices (telemedicine devices)are present and capable of navigating to the selected patient, the RPImay automatically determine an optimal endpoint device to dispatch tothe patient.

Factors involved in determining the optimal endpoint device could bedistance or estimated travel time to the selected patient or remainingbattery life of respective telepresence devices. Additionally, the RPImay dispatch the endpoint device that can reach the selected patientwhile minimizing travel through areas with poor wireless connectivity,confined spaces/areas, high-traffic areas, sensitive and/or secureareas, dangerous areas, and/or otherwise undesirable zones.

The endpoint list may also include doctors, nurses, staff, or any otherpersons that may currently (or for a scheduled period of time) beassociated with a particular location and/or patient to which theendpoint can navigate. The endpoint list may be searchable and/orfilterable. In some embodiments, the endpoint list may be implementedwith a text box, in a window, or in separate tabs. As the user entersalphanumeric characters into the text box, the list may beinstantaneously filtered to exclude endpoints whose names do not matchthe character string currently contained in the text box.

Other filtering parameters may be specified, such as endpoint type,manufacturer, status, facility, building, floor, room, customer, and/orany other grouping. Logical, arbitrary, or otherwise customizedgroupings of endpoints may be created by a user or administrator, andthese groupings may additionally be used to filter or otherwise searchthe list of endpoints. In some embodiments, each endpoint in the listmay have an associated status indicator, which informs the user whethera device is ready, busy, offline, in a private session, and/or in amulti-presence session (which the user may join to receive sessionaudio, video, images, or potentially control the some or all functionsof the endpoint).

FIG. 5 illustrates an embodiment 500 of a PED 505 displaying aconnection wizard page as a user is connected to a telepresence networkand/or telepresence device. The connection wizard may facilitate theconnection of a user to a telepresence network, a specific telepresencedevice, and/or other operators of RPIs. In various embodiments, thedisplayed connection wizard page may be purely a placeholder imagedisplayed while a connection is established. In other embodiments, theconnection wizard may enable or allow a user to select variousconnection parameters and/or settings.

FIG. 6 illustrates an exemplary embodiment 600 of a module within theRPI configured to provide visual and interactive control of atelepresence device. As illustrated, the RPI may be divided into anupper panel 610 and a lower panel 620. The upper panel 610 may include avideo window 615 showing the user of the PED 605 what the camera 680 iscapturing. The upper panel 610 may display a video feed originating froma camera on the remote presence device. The upper panel 610 may furtherinclude various informational items and/or control panels, described ingreater detail in conjunction with subsequent figures. The lower panel620 may be used to display additional information and/or to receiveinputs from a user of the PED.

FIG. 7 illustrates an embodiment 700 of an RPI on a PED 705 including amap 720 and a navigation module 710. The map 720 may display a plan viewmap of a healthcare facility. An image captured by the camera 780 may bedisplayed in a window 715. A list of destinations 730 may be displayedin an upper panel. According to various embodiments, a user may select adestination by selecting a room within the healthcare facility, or byselecting a patient within a patient tab. The selection of a room may beconsidered a navigation input. The selection may then be translated intoone or more navigation instructions to guide the telepresence robot,autonomously or semi-autonomously to the selected location.

Once a destination, patient, or other person is selected, the RPI maycommunicate the desired destination to the telepresence device vianavigation instructions. The navigation instructions may allow for thetelepresence device to be manually navigated, semi-autonomouslynavigated, or autonomously navigated to the desired destination. Systemsand methods for manual, semi-autonomous, and autonomous navigation aredescribed in U.S. patent application Ser. No. 13/360,579, previouslyincorporated by reference.

FIG. 8 illustrates an embodiment 800 of a notification 820 generated orreceived by the RPI. The notification 820 may be displayed on a PED 805in a lower panel or an upper panel. As illustrated, the notification 820may include the remaining battery life of the telepresence robot, therobot number, the location of the robot, and/or the local time. As inprevious embodiments, a window 815 may show the image currently beingcaptured by a camera. An upper panel 810 may be maximized, or shown in afull-screen mode, by selecting a full-screen icon 830. In someembodiments, the notifications 820 may be displayed as overlays when theupper panel 810 is maximized or is in a full-screen mode.

The RPI may be designed to provide a user with notifications regardingvarious states of a device (e.g., the PED 805 or the telepresencedevice), a connection, or a session. For instance, the RPI may displayindications of whether video or audio recording is active and/or enabledor if a laser pointer or remote pointer is active, a battery level orremaining time available based on the current battery level, related tonetwork performance, and/or a wireless signal strength.

In one embodiment, the RPI may provide a message to the user (who may bemoving around) if the PED loses (or detects a decrease in strength of) acommunication signal (e.g., a WiFi or 4G signal) as the result of theuser's movement and/or as a result of the telepresence device leaving agood WiFi zone, or some loss, outage, or failure elsewhere in the link.The message provided to the user may distinguish whether the loss,outage, or drop in strength has occurred locally, remotely, or elsewherein the link.

In some embodiments, the RPI may be configured to provide notificationsor alerts using various display modifications, annotations, or overlays.For example, the screen may pulsate with a partially translucent overlayto indicate that a battery of the telepresence device is dying and/orthat connectivity is below a predetermined level. For example, thescreen may pulsate with a red glow, possible with a vignette such thatit is increasingly translucent toward the center of the display, toindicate that a battery is dying.

In one embodiment, a notification may include one or more pop-updialogue boxes to bring critical information to the attention of theuser of the RPI. For example, if the connectivity falls below a levelconsidered clinically viable, the RPI may request that a user of the RPIacknowledge the current limitation in order to continue the telepresencesession. As another example, the RPI may request that a user acknowledgeanomalous conditions associated with the RPI, the patient, and/or thetelepresence device. The RPI may request that a user acknowledge thatthey are aware of or have noticed patient conditions considered outsideof predetermined ranges. For example, the telepresence device and/or theRPI may recognize that a patient's heart rate is below a threshold leveland request that a remote user of the RPI acknowledge awareness of thecondition.

FIG. 9 illustrates an embodiment 900 of a media manager module 920 ofthe RPI configured to allow a user to capture and manage audio and/orvisual media from the telepresence device. As illustrated, the upperpanel 910 of the PED 905 may include a live video feed from thetelepresence device. The live video feed in the upper panel 910 may beexpanded to full-screen via the full-screen icon 930. The lower panelmay include a media manager module 920 adapted to capture still images,audio, video, and/or a combination thereof from the telepresence device.Additionally, the media manager module 920 may allow for editing,playback, trimming, storing, emailing, and/or other functions.

For example, when a user taps the video or movie projector icon in atoolbar, the RPI may display the native video controls of the PEDoverlaid above the live video feed, along a toolbar across the upperpanel 910 or a lower panel, or elsewhere on the screen. Additionally, alibrary of network-accessible or locally stored video clips or moviesmay be made accessible in the lower panel. The user may navigate thevideo clip library by swiping left or right, which may cause successiveclips to be visually cycled from completely obscured to partiallyobscured to fully visible (and selectable/playable), to partiallyobscured again, and so on. Once the user locates a desired video, theuser may drag and drop the video from the media manager module to theupper panel 910 to cause the video to be displayed on a screen on thetelepresence device.

The RPI may enable playback to be controlled from the PED and/or thetelepresence device. Once a video is selected, the RPI may enable theuser to navigate or skip through an index of key frames contained in thevideo. The RPI may additionally enable the user to delete a video usinga flicking gesture. For security and privacy, videos may be stored in anencrypted format. In another example, when the user taps the camera iconin a toolbar, the interface may display a cover-flow image library inthe lower pane. Images may be searched, browsed, displayed on the PED,or displayed at the remote endpoint in the same manner as describedabove with respect to video files in the media manager.

FIG. 10 illustrates an embodiment 1000 of menu 1040 of the RPI allowinga user to modify various settings on the local PED 1005 and/or a remotepresence device. As illustrated, a window 1010 may show the user what iscurrently being captured by the camera 1080. The upper panel 1015 maydisplay a live video feed generated by the telepresence device. The menu1040 in the lower panel 1030 may contain settings for local brightnesscontrol 1045, remote brightness control 1065, microphone levels 1035,speaker levels 1055, and/or any of a wide variety of other features.

A lower bar 1020 may also include one or more management tools. Asillustrated, a picture-in-picture selection may allow a user toselectively disable the picture-in-picture window 1010. Anauto-brightness feature 1045 may be enabled via a checkbox. The controlsettings may further include a microphone level meter 1050 to indicate avolume or sound pressure level relative to a maximum specified input orclipping level. Additionally, the settings control may include amicrophone gain slider 1035 to allow adjustment of a microphone gain toa desired level. Similarly, a speaker level meter 1060 may graphicallyillustrate the current volume or sound pressure level relative to amaximum specified output or slipping level. The remote controls mayinclude a picture-in-picture checkbox to enable the user to toggle apicture-in-picture display of the remote video view at the remoteendpoint.

FIG. 11 illustrates an embodiment 1100 of the RPI in full-screen videomode. As illustrated, a window 1115 may show a picture-in-picture of theimage captured by camera 1180. The upper panel 1110 may display afull-screen live video feed captured by a telepresence device.

FIG. 12 illustrates an embodiment 1200 of the RPI in full-screen datamode. In the illustrated embodiment, the image captured by camera 1280may not be displayed in a picture-in-picture window to make more roomfor data entry and/or visualization. In some embodiments, the imagecaptured displayed by the telepresence device that is normally the imagecaptured by the camera 1280 may display a no signal or screensaverimage, or a previous image of the healthcare practitioner may be“frozen” on the screen.

The PED 1205 may allow data to be entered via the touch screen usingkeyboard 1230. As illustrated, the RPI may display a single data panel1215 in a full-screen mode by removing the upper panel typically used todisplay the live video feed from the telepresence device. Various tabs1210 may toggle the screen between various data input and/orvisualization modes. In some embodiments, the remote video view maycontinue to be displayed in a small window or panel.

FIG. 13 illustrates an embodiment 1300 of the RPI including a visualrepresentation of patient telemetry data 1350 displayed in a lower panel1320. The PED 1305 may again include a camera 1380 and a window 1310 maybe displayed by the RPI to show a user what is currently being capturedby the camera 1380. As illustrated, full-screen icons 1335 may beavailable in the RPI to transition either of the lower panel 1320 or theupper panel 1315 to a full-screen mode. A software button 1330 may allowthe telemetry data to be toggled, changed, scrolled, and/or otherwisemanipulated. A patient icon 1325 may allow a user to select atelepresence device and/or telemetry data associated with a differentpatient. The telemetry data 1350 may be displayed as numerical valuesand/or in graphical form 1340.

FIGS. 14A-C illustrate exemplary embodiments of a telepresence device.As illustrated in each of FIGS. 14A-C, a telepresence device maycomprise a base 1410 capable of being manually driven, semi-autonomouslydriven, and/or autonomously driven. Various display features, connectionfeatures, and/or data ports 1420, 1430, and 1465 may be available on amid-section of the telepresence device. The telepresence device may alsoinclude a handle 1435. A head portion 1440 of the telepresence devicemay include one or more cameras 1460, speakers, and/or microphones. Inaddition, the head portion 1440 may include one or more two- orthree-dimensional depth sensors, such as LADARs, LIDARs, or structuredlight projectors/scanners. Multiple cameras 1460 may be useful to render3D images, and multiple microphones and/or speakers may be useful forrendering and/or generating directional sound. The head portion may alsoinclude a display 1450 configured to display, for example, an imagecaptured using an RPI on a PED.

The display 1450 and/or the interface 1430 may comprise a touch screenor other interface to receive inputs. In some embodiments, if atelepresence device is an autonomous mobile telepresence device, thedisplay 1450 and/or the interface 1430 may provide a list ofdestinations, healthcare practitioners, and/or patients to which, asdescribed herein, the telepresence device can be sent or connected. Thedisplay 1450 and/or the interface 1430 may also enable a person to stopthe telepresence device when it is autonomously navigating and,likewise, enable the telepresence device to resume autonomous navigationto its destination. The display 1450 and/or the interface 1430 mayadditionally have a button or menu option that instructs thetelepresence device to autonomously navigate to an out of the waylocation (e.g., a wall, corner, etc.), a dock, a storage location,and/or a charging station. The display 1450 and/or the interface 1430may include buttons or menu options for various settings or to page ornotify support personnel of a problem with or question regarding theoperation of the telepresence device.

FIG. 15 illustrates a panel 1530 including a plan map view, a video feedwindow, and various settings panels. The panel 1530 may be displayed invarious forms and configurations via an RPI on a PED, such as the PEDs1510, 1520, and 1540. Each of the PEDs 1510, 1520, and 1540 illustrateexamples of panel arrangements generated by the RPI to maximize theuseable screen space for various tasks performed using the RPI. It willbe appreciated by one of skill in the art that a larger display mayaccommodate more panels or larger panels. Similarly, devices withmultiple displays may accommodate one or more panels on each display.For example, a desktop version of the RPI may utilize multiple monitorsconnected to the desktop to display one or more panels.

FIG. 16 illustrates an embodiment 1600 of an RPI on a PED 1605. The RPImay utilize the camera 1680 to transmit an image of the user to atelepresence device. The image captured by the camera 1680 may bedisplayed in a picture-in-picture window 1610. The upper panel 1615 ofthe RPI may display a live video feed received from the telepresencedevice. The lower panel 1620 may display a plan map view. In someembodiments, the plan map view may show a current location of thetelepresence device within a healthcare facility (or other locale). Theuser may manually navigate the telepresence device using the live videofeed and the plan view map. Alternatively or additionally, the user mayselect a location within the plan view map and the telepresence devicemay autonomously or semi-autonomously navigate to the selected location.

The input intended to direct the telepresence device to a new locationmay be considered a navigation input. For example, the selection of aroom, patient, healthcare practitioner, location within a video feed,and/or other selection or input intended to cause the telepresence robotto navigate may be considered a navigation input. The navigation inputmay then be translated into one or more navigation instructions to guidethe telepresence robot, autonomously or semi-autonomously, to theselected location.

FIG. 17 illustrates an embodiment 1700 of an RPI on a PED 1710displaying multiple panels. As illustrated, a radiography panel 1720 maydisplay images associated with a patient displayed in a live video feed1750. Telemetry data 1730, lab results 1740, patient data 1760, andphysician notes 1770 may be displayed in various other panels on the PED1750 via the RPI. In a multi-user telepresence conference each of theparticipating users may be displayed in a panel 1790. According tovarious embodiments, each of the panels 1720, 1730, 1740, 1750, 1760,1770, and 1790 may be moved, enlarged, merged with another panel,removed, and/or captured (recorded), intelligently based on decisionsmade by the RPI, based on usage history, based on relevancy, and/orbased on user selection. A camera 1780 may be selectively enabled ordisabled by the user.

The RPI may enable complete integration of patient data monitoring withthe remote telepresence session, thereby adding a dimension ofdata-driven functionality uniquely valuable in telepresenceapplications. The user may select an icon from a toolbar or other panelto activate a patient bedside data monitoring app, such as those offeredby any of a variety of real-time patient data monitoring applicationproviders. Upon selecting the appropriate icon, a patient datamonitoring window may appear in the RPI. The user may expand this paneto a full-screen view, reposition the pane, and/or resize the pane asdescribed above. The RPI may show any number of real-time or archivedpatient biometrics or waveforms, such as temperature, heart rate, pulse,blood pressure, oxygen saturation, etc.

Using the touch-screen interface, the user may pause and resumereal-time, time-delayed, or archived patient data. The user may moveback and forth through time-based patient data using dragging or swipinggestures, or the user may zoom or scale the waveform or metric along anamplitude axis and/or time axis. The application may further allow theuser to set markers along a waveform to measure variations in amplitudeor time associated with various features of the patient data, such aspeaks, valleys, maxima or minima (global or local), global averages,running averages, threshold crossings, or the like.

The data may be collected from bedside monitors or other monitoringdevices in real-time and archived for a period of time (or indefinitely)in a server or database. The monitoring app may be a separateapplication and/or integrated within the RPI. The monitoring app mayretrieve the relevant data and provide it to the RPI through anapplication programming interface (API) and/or the RPI may independentlyretrieve the data from a database.

The data may also be collected by the telepresence device by, forexample, directing a camera of the telepresence device to the display ofa monitoring device, and either recording video of the monitor displayor performing image analysis on the video image to extract the patientdata. The user and/or telepresence device may annotate the data andstore the annotations with the data, either locally or in a remoteserver, for later retrieval. The monitoring app may enable alarms,alerts, notifications, or other actions or scripted activities set totake place in response to certain events in the data.

Further, the interface may integrate available ADT with patient bedsideor biometric data. For example, if a patient's vitals or otherbiometrics trigger an alert or alarm condition, the telepresence devicemay be configured to autonomously navigate to the bed or room number ofthat patient, and send a notification or invitation to a doctor,caregiver, or specialist to begin a telepresence session with thepatient. Additionally, when a healthcare practitioner initiates asession with a telepresence device and selects either a location,destination, or patient to visit with the autonomous telepresencedevice, the bedside or biometric data for a patient associated with theselected location, destination, or patient may be automaticallyretrieved and used to populate a “dashboard” of patient data that thehealthcare practitioner can then review, annotate, or otherwise interactwith as discussed above and depicted in FIG. 17.

Moreover, an autonomous mobile telepresence device may be used toconduct patient rounds in a healthcare facility. As the telepresencedevice moves from one location to the next, the location of thetelepresence device may be used to retrieve the name and/or other dataof a patient(s) associated with that location. For example, thetelepresence device may retrieve patient biometrics, bedside data,electronic medical records, and/or other patient data to populate apatient dashboard on a display of the PED. In one embodiment, thisinformation may be retrieved from a health level 7 (HL7) compliantserver associated with the facility, healthcare practitioner, and/orpatient.

In addition, an autonomous mobile telepresence device may be scripted orscheduled to make scheduled stops at various beds, rooms, locations, orpatients. The RPI may retrieve the names or contact info of people (suchas doctors, nurses, students, family members, etc.) associated with ascheduled or upcoming stop at a particular patient or location, and senda notification via SMS, email, etc., to the associated people invitingthem to join the telepresence session by receiving audio and/or videofrom the session on a PED via the RPI.

To accommodate a time interval that may be necessary or convenient toallow others to join the session, the telepresence device may sendinvitations, notifications, and/or reminders to join the session apredetermined amount of time prior to the time the session is scheduledto begin. Repeated or reminder notifications may be sent to each partyat regular or decreasing intervals to remind them of an upcomingsession. The notifications may contain a hyperlink to follow to join thesession, a link to the RPI, an app notification or badge for display onthe PED, or the address or phone number of another device address toconnect to. The notification may further include a username, password,pin and/or other credential(s) that the invitees may provide to join thesession. The length of the session may be at least partially based onthe number of users connected and/or their priority levels.

FIGS. 18A and 18B illustrate an embodiment 1800 of what may be displayedduring a consult on a telepresence device 1850 and via an RPI on a PED1805, respectively. As illustrated, the telepresence device 1850 mayinclude audio and/or visual equipment 1880 to capture images and/oraudio for display on the PED 1805. PED 1805 may include a camera tocapture an image 1815 for display on a screen 1857 of a head portion1855 of the telepresence device 1850. A lower portion of thetelepresence device may include adjustment knobs for the microphonevolume 1861 and/or the speaker volume 1862. Additionally, a screen 1870may provide additional information about the user of the PED 1805.Accordingly, a patient being cared for via the telepresence device 1850may see a healthcare practitioner using the RPI on the PED 1805.Similarly, the healthcare provider may see and interact with the patientvia the telepresence device using the RPI on the PED 1805.

FIG. 19 illustrates an embodiment of a telepresence device 1900including audio and/or visual instruments 1980, controls 1961 and 1962,and displays 1950 and 1970. In a screensaver mode, the upper display1950 may preserve the screen by displaying information about a company,the healthcare facility, medical facts, and/or other informationassociated with healthcare in general. A lower display 1970 may alsoenter an independent screen saver mode, and/or allow for user inputsassociated with the telepresence device and/or the information displayedvia the upper display 1950.

FIG. 20 illustrates various embodiments of avatars 2091, 2092, 2093, and2094 and/or personalities that may be assigned or used by a healthcarepractitioner and/or telepresence device. For instance, while the image2090 displayed on the display 2050 may normally be associated with theimage captured by a camera of a PED via the RPI, any of a wide varietyof characters, avatars, cartoons, licensed caricatures, and/or otherimages may be used in place of an image received from the camera on thePED. The avatars 2091, 2092, 2093, and 2094 may be particularly usefulto give human-like traits to the telepresence device. The telepresencedevice may, as previously described, include controls 2061 and 2062 foraudio and a touch sensitive screen 2070.

FIG. 21 illustrates an RPI that can be displayed on a PED includingvarious informational, control, video, and settings panels 2120, 2130,2140, 2150, 2160, 2170, 2180, and 2190. As illustrated, a live videofeed may be displayed in a panel 2110. Remote brightness, zoom, andvolume controls may be accessible via a panel 2190. Media managementcontrols may be accessible via a panel 2180. Advanced controls withvarious tabs 2160 may be available via a panel 2170. A current imagecaptured by a camera on the PED may be displayed in a panel 2150. Localzoom, brightness, volume, and microphone controls may be accessible viaa panel 2140. Additional controls and tabs of control icons may beaccessible via a panel 2130.

In the illustrated embodiment, a user of the RPI may select a navigationmode via a selection panel 2120. Inputs for selecting the navigationmode and/or inputs to direct the actual navigation may be performedusing any of a wide variety of inputs, including, but not limited to, avoice input, a keyboard, a mouse, a touch input, a stylus, and/or viaanother peripheral input device. In each of the navigation modes 2120, auser may provide a navigation input in one or more manners. Thenavigation input may then be translated (processed, lookup table, etc)to generate and transmit navigation instructions to the telepresencerobot to guide the telepresence robot, autonomously orsemi-autonomously, to a desired location.

FIG. 22 illustrates an embodiment 2200 of an RPI configured to utilizedestination-based navigational modes for navigating a telepresencedevice. As illustrated, a centralized panel 2210 may display a livevideo feed 2220. Various remote settings controls may be available in apanel 2290. Another panel 2240 may provide similar audiovisual settingcontrols for a PED. A window 2250 may display the image currentlycaptured by a camera on the PED. Advanced controls may be available in apanel 2270, and multiple tabs 2260 may provide for additionalinformational and/or control panels. The illustrated embodiment 2220includes a destination navigation panel 2230 configured to allow a userto select from a list of destinations. Once a destination is selected,the RPI may facilitate communication with the telepresence device todirect the telepresence device to the selected destination. Thetelepresence device may be manually driven to the selected destinationor may autonomously (or semi-autonomously) navigate to the selecteddestination.

According to various embodiments, the destinations may include variouslocations within a healthcare facility, specific patient names, and/orthe names of various healthcare practitioners. In some embodiments, thelocations of healthcare practitioners may be determined using cameraswithin the healthcare facility, global positioning systems, localpositioning systems, radio frequency identification (RFID) tags, and/orthe like.

FIG. 23 illustrates an embodiment 2300 of an RPI including a panel 2320displaying a live feed from a telepresence device. A lower panel 2310may display a list of patients. A user may select a patient on thepatient list to direct the telepresence device to navigate to theselected patient. Again, the telepresence device may be manually driven,autonomously navigate, or semi-autonomously navigate to the selecteddestination.

FIG. 24 illustrates an embodiment 2400 of an RPI including panel 2410 ofa plan view map allowing for map-based navigational modes for navigatinga telepresence device. As illustrated, the map-based navigational modemay be part of an advanced control panel including multiple tabs. Thesize of the panel 2410 may be adapted to suit a particular need and maybe user-selectable. Media controls 2440 may allow a user to captureaudio and/or visual information from the live video feed in panel 2420.A panel 2430 may display the current image captured by a camera of a PEDrunning the RPI. As illustrated the panel 2430 displaying the currentimage captured by the camera of the PED may include integrated settingscontrols. Various tabs 2450 may provide access to additional features,options, help, and/or navigation within the RPI.

FIG. 25 illustrates an embodiment 2500 of a full-screen view of ahallway from a telepresence device as visualized on a PED via an RPI.The panel 2510 may display a live video feed from a telepresence device.Icons 2520 and 2522 may be overlaid on the video feed and may provideaccess to various controls or settings. A destination panel 2530 mayalso be overlaid on the video feed and allow a user to select adestination. As in previous embodiments, the RPI may communicate aselected destination to the telepresence device, which may autonomouslyor semi-autonomously navigate to the selected destination.

FIG. 26 illustrates an embodiment 2600 of a full-screen view of a livevideo feed from a telepresence device as may be displayed via an RPI.Various icons 2620 and 2622 may provide access to various features,settings, options, and/or other aspects of the RPI. The icons 2620 and2622 may be overlaid on the live video feed or a separate panel may bedisplayed containing icons 2620 and 2622.

FIG. 27A illustrates another embodiment 2710 of a live video feed from atelepresence device, as may be displayed via an RPI. An arrow 2730 maybe overlaid on the video feed as a navigation input to indicate anintended or desired navigation path of a telepresence device. The arrowmay be drawn by the RPI after receiving the live video feed or by thetelepresence device prior to sending the live video feed. In someembodiments, the user may draw the arrow 2730 to indicate a desireddirection of travel. The arrow 2730 may then be translated by the RPI orby the telepresence device into directional commands to navigate thetelepresence device. Again, various icons 2720 may provide access tovarious features, settings, options, and/or other aspects of the RPI.The icons 2720 may be overlaid on the live video feed or a separatepanel may be displayed containing icons 2720. The arrow 2730 may be avector quantity describing both the direction and the velocity (current,planned, or directed) of the telepresence device.

The arrow 2730, provided as a navigation input, may be used to generatenavigation instructions that are transmitted to a remote presencedevice. As illustrated, the navigation input may be in the form of avector (arrow 2730) drawn as either a final endpoint selection or as avector including a beginning point and an end point. The end point ofthe vector may represent the location to which the telepresence deviceshould navigate. Accordingly, navigation instructions may be generatedbased on the current location of the telepresence device and theendpoint of the vector within the live video feed. In some embodiments,the vector's image pixel endpoint may be mapped via a lookup table toone or more translate and rotate commands to cause the telepresencedevice to navigate to the selected location. In some embodiments, thelength of the vector may correspond to the desired distance to move thetelepresence device, a desired acceleration of the telepresence device,and/or a desired velocity of the telepresence device.

As previously described, a navigation input may direct a telepresencedevice to navigate forward (with respect to the live video feed) 3meters and to the right 2 meters. Any of a wide variety of navigationinstructions may be possible to correctly navigate the telepresencedevice. For instance, the navigation instructions may direct thetelepresence device to navigate approximately 3.6 meters at a 34 degreeangle relative to the video feed. Alternatively, the navigationinstructions may direct the telepresence device to navigate 3 metersforward and then 2 meters to the right. As will be appreciated, anynumber of navigations routes and corresponding navigation instructionsmay be possible. The navigation input may be translated into navigationinstructions as a plurality of drive forward commands coupled withrotate commands.

In some embodiments, the live video feed may be delayed slightly due tonetwork and/or processing limitations. This video latency may result ininaccurate navigation instructions. Accordingly, the navigation inputmay be adjusted based on the known video latency and the current or pastvelocity and direction of the robot. For example, the selection of alocation within the video feed may be translated into navigationinstructions that compensate for the latency of the video feed. Forinstance, if an end point (endpoint pixel) of a vector drawn on thevideo feed maps to a position 1 meter forward and 0.5 meters to theright relative to the current location of the telepresence device, andthe telepresence device has moved 0.2 meters forward since the videoframe on which the vector was drawn was captured, or since theassociated command was issued, a lookup table entry for 0.8 metersforward and 0.5 meters to the right may be used to determine and/oradjust the navigation instructions. Video latency calculation andcommand compensation may be performed at the telepresence device, at aremote interface device, and/or at any networked computing device. Insome embodiments, the navigation instructions may be generated tocompensate for the video latency. In other embodiments, the telepresencedevice and/or other computing device may adjust the navigationinstructions to compensate for the video latency.

FIGS. 27B-27H illustrate various aspects of a mouse-based drivinginterface 2711 of an RPI for manually or semi-autonomously controlling atelepresence device. As illustrated in FIG. 27B, the mouse-based drivinginterface 2711 for navigating a telepresence device via an RPI may bepresented within a live video feed 2712 from the telepresence device. Afour way controller 2750 may be overlaid on the live video feed.According to various embodiments, the arrows of the four way controller2750 may be selected by a finger or stylus on a touch screen device orvia a pointer 2751 on a touch screen device or as controlled by aperipheral device (e.g., a keyboard or mouse). For instance, a user mayoperator a mouse to control the pointer 2751 and click on one of thearrows on the four way controller 2750.

In some embodiments, the live video feed 2712 and the four waycontroller 2750 may be displayed in a full-screen or expanded mode. Inother embodiments and/or modes, various additional panels, icons, tabs,and/or other objects 2713, 2714, 2715, 2716, 2717, 2718, and 2719simultaneously with the mouse-based driving interface. For example, aremote controls panel 2713 may allow a user to control various settingsof the remote presence device, such as the audio and video settings. Amedia control panel 2714 may allow a user to open, play, record, and/orotherwise manipulate the current live video (or audio) feed 2712,network-accessible audiovisual material, remotely (at the telepresencedevice) stored audiovisual material, and/or locally stored audiovisualmaterial.

An advanced controls panel 2715 may allow a user (or select users basedon account privileges) to modify various aspects of the connection orsession settings, access external or peripheral applications (e.g.,StrokeRESPOND), and/or control other aspects of the remote telepresencesession. A local controls panel 2716 may allow a user to control varioussettings of the RPI, the PED, and/or other local peripheral devices,such as the audio and video settings associated with the telepresencesession. A battery life indicator 2717 may provide information regardingthe current battery life and/or expected battery life based on currentor past consumption rates. Various additional icons and tabs 2718 and2719 may provide additional controls and/or options within the RPIand/or associated with the remote presence device.

FIG. 27C illustrates the mouse-based driving interface 2711 with longdrive vector 2755 drawn on the video feed to indicate a direction and arelatively fast velocity. According to various embodiments, thedirection of the vector may be associated with an intended, directed (asdirected by the user of the RPI), and/or current direction of travel. Invarious embodiments, a user may click the uppermost arrow of the fourway controller 2750 and then drag a vector to describe a desireddirection of travel. The length (magnitude) of the vector may correspondto the velocity and/or acceleration of the telepresence device.Additionally, a grid display 2760 may illustrated the current directionof travel and/or the velocity (magnitude) of the telepresence device ona grid. The grid may correspond and/or display travel angles (e.g., indegrees, radians, with respect to north, etc.) and/or a velocity as anumeric or descriptive value. For example, the length of the dark lineon the grid display 2760 may be aligned with numbers between 1 and 10 ordescriptive terms like “slow,” “medium,” and “fast.”

FIG. 27D illustrates the mouse-based driving interface 2711 with a shortdrive vector 2756 drawn on the video feed to indicate a direction and arelatively slow velocity. A corresponding display of a short line may bedisplayed in the grid display 2760.

In various embodiments, a navigation input in the form of a vectorprovided in a mouse-based vector input mode may be translated intonavigation instructions through the use of a lookup table. In someembodiments, the lookup table may include values that compensate forlatency. That is, the navigation instructions returned by the lookuptable may be based on the navigation input and the current or recentaverage video latency, as described herein.

FIG. 27E illustrates the mouse-based driving interface 2711 with avector 2757 for rounding a corner 2772 within the RPI. According tovarious embodiments, the RPI and/or telepresence device may detectobjects such as the walls to determine passable hallways and corridors.Alternatively or additionally, the RPI and/or telepresence device mayutilize maps to determine that a vector 2757 is intended to direct atelepresence device to turn down a different hallway or round a corner.As illustrated, the vector 2757 in FIG. 27E may be intended to roundcorner 2772. In some embodiments, the user may draw a rounded or curvedvector 2757. In other embodiments, the RPI and/or telepresence devicemay interpret a straight vector 2757 as intended to direct thetelepresence device down the hallway.

FIG. 27F illustrates the mouse-based driving interface 2711 with avector 2758 drawn to cause the telepresence device to spin in place.According to various embodiments, a horizontal vector 2758 (relative tothe live video feed 2712) may be used to direct the telepresence deviceto spin in place either clockwise or counterclockwise depending on thedirection of the horizontal vector 2758.

FIG. 27G illustrates the mouse-based driving interface 2711 with avector 2759 drawn towards an object (a wall). According to variousembodiments, the mouse-based driving interface may prevent thetelepresence device from colliding with objects (such as walls, people,and other objects). As previously described, an obstacledetection/obstacle avoidance (ODOA) system may prevent such collisions.In some embodiments, a user may deactivate the ODOA system, such thatthe user may operate in a “nudge mode” and continue to slowly move thetelepresence device despite the telepresence being in contact with orclose proximity to another object. This mechanism may automaticallytime-out such that the ODOA system is re-engaged after a period ofinactivity.

Placing the telepresence device in nudge mode may cause a camera of thetelepresence device to be automatically positioned to look down aroundthe body of the telepresence device and either in the direction of drivecommand issued by the user or in the direction of an object closest toor in contact with the telepresence device (so that the user may seewhat obstacle he or she is commanding the telepresence device to nudge).Alternatively, the ODOA system may be entirely deactivated. In oneembodiment, the ODOA system may be deactivated so long as thetelepresence device is driven or moved below a specified velocity.

FIG. 27H illustrates the mouse-based driving interface 2711 used toreverse the telepresence device 2785 with a camera of the telepresencedevice (or other associated camera) oriented in reverse and slightlydownward toward the floor 2790. In some embodiments, by selecting (touchor mouse click) the bottom arrow of the four way controller 2750, thetelepresence device 2785 may be reversed. The reverse mode may allow forvector controls (direction and velocity) or may alternatively reversestraight back and at a constant velocity. In some embodiments, a rearcamera may be displayed on the live video feed 2712 to show thedirection of travel and help prevent the telepresence device 2785 fromcolliding with objects. In other embodiments, a head portion of thetelepresence device may be rotated to the direction of travel (reverse)and/or inclined slightly downward to facilitate the reverse navigation.

In various embodiments of the mouse-based driving interface, thecontrols and vectors may be “drawn” as overlays on the live video feed(e.g., via a click and hold of a mouse button or a tap and hold via astylus or finger). In other embodiments, a panel or peripheral devicemay be used to “draw” the vectors to control the velocity and/ordirection of travel. In some embodiments, the velocity selected by thelength of the vector may be overridden based on other factors (such asobstacle detection, congestion, human presence, etc.).

According to various embodiments, a head portion and/or the camera ofthe telepresence device may be re-centered via an icon within themouse-based driving interface 2711 or via a keyboard (or otherperipheral device) selection. In some embodiments, the selection of anyof the arrows within the four way controller 2750 may orient orre-orient the head portion of the telepresence device.

FIG. 28 illustrates an embodiment 2810 of full-screen live video feedfrom a telepresence device including a bedside view of a patient bed2820 and an associated patient monitor 2830. As illustrated, the livevideo feed may be able to visualize a patient in a patient bed 2820 aswell as a patient monitor. In some embodiments, the RPI and/or thetelepresence device may detect a refresh rate of the patient monitor2830 and dynamically match the refresh rate to reduce scrolling barsand/or flickering. In other embodiments, the live video feed of thepatient monitor 2830 may be digitally enhanced to reduce the flickeringand/or scrolling bars.

FIG. 29 illustrates an embodiment 2910 of an RPI including aclick-to-zoom feature 2925. The click-to-zoom feature 2925 may be usedto zoom in on a specific area of a live video feed, a captured videofeed, or a still image. The box defining the click-to-zoom feature 2925may be drawn as a box, or by dragging one or more corners. Depending onthe PED used, the RPI may receive the click-to-zoom box using any of awide variety of input devices, such as via a touch, a stylus, or a mousepointer.

FIG. 29 also illustrates a multi-party session. In the illustratedembodiments, multiple users are shown in a session guests panel 2930.The session guests panel 2930 may include images captured by PEDsassociated with each of the guests. Accordingly, multiple users mayparticipate simultaneously. In such embodiments, one or more of theusers may have limited access and/or control of the telepresence device.As in previous embodiments, control and options icons 2940 and 2942 maybe overlaid and provide access to various features of the RPI.

FIG. 30 illustrates an embodiment 3000 of a PED running an RPIconfigured to display a live video feed in an upper panel 3015 alongwith a current image window 3010. As illustrated, the RPI mayincorporate sub-applications and/or provide access to relatedapplications, such as a StrokeRESPOND application (in a lower panel3020) configured to provide one or more functions and/or workflowprocesses associated with strokes. The StrokeRESPOND application maydisplay various patient names 3030, which may be filterable, at 3025,and allow a user to select them, at 3035, in order to see more details.The RPI may allow for any number of sub-routines, sub-applications,and/or other applications to run within the RPI or be launched from theRPI. As other examples, the RPI may provide access to a dictionary, amedical text, an internet search engine, and/or any other external orintegrated application.

In some embodiments, while an application is open in the lower panel3020, the user may switch to viewing the application in full-screen modeby grabbing the upper part of the application window and dragging upwardtoward the upper panel 3010. The user may return to the two-pane view bydragging downward from the top of the upper panel 3010. The full-screenapplication view may include a picture-in-picture of the live video feedfrom the telepresence device so that the user may continue to monitorthe remote environment. Some applications (and/or the RPI) may continueto run in the background, even when not displayed.

FIG. 31 illustrates an embodiment 3100 of an RPI on a PED 3105, in whichan image 3125 captured by a camera of the PED 3105 is displayed as atransparent image overlaid on a live video feed 3110 originating from atelepresence device. The transparent image 3125 may alternatively bedisplayed in a lower panel 3120.

FIG. 32 illustrates an embodiment 3200 of an RPI on a PED 3205,including a toolbar 3225 in a lower panel 3220. The toolbar may providequick access to any of a wide variety of settings and or features of theRPI. A user may select an icon using any of a wide variety of methodsdepending on the PED. For instance, a user may touch an icon to selectit. Settings and/or features of the RPI may be accessed simultaneouslywhile a live video feed is shown in the upper panel 3210. A mediamanagement toolbar 3230 (selectively enabled) may allow for the videofeed in upper panel 3210 to be recorded, at 3240. A notification 3235may alert a user of the PED 3205 that the battery on the telepresencedevice is nearly depleted. As in previous embodiments, a window 3215 maydisplay the image currently being captured by a camera on the PED 3205or managing modules and control operations available via an RPI, whilesimultaneously displaying a video feed from a telepresence device.

According to various embodiments, the toolbar 3225 may provide access toa handset, a stethoscope, a camera, a video, a live cursor, a laserpointer, microphone settings, a map, navigational options, a disconnectbutton, and/or other features, options or settings. The toolbar 3225 mayprovide access to various other functions or applications, such asStrokeRESPOND, SureNotes, a media manager, patient data, lab results,image data, and/or team communication.

FIG. 33 illustrates an embodiment 3300 of an RPI on a PED 3305 thatincludes a media management toolbar 3325 associated with a mediamanagement window 3330 in a lower panel 3320 of the PED 3305. Asillustrated, an upper panel 3315 may include a live video feed of apatient. The user of the RPI may access stored videos, images, audio,and/or telemetry data associated with the patient via the mediamanagement toolbar 3325.

FIG. 34 illustrates an embodiment 3400 of an RPI on a PED 3405 includinga video window 3410 displaying a list of telepresence devices to whichthe user has access, a work space window 3430 displaying a list ofpatients, and a toolbar 3415 as a tool belt dividing the display. Invarious embodiments, the selection of a telepresence device via videowindow 3410 will display a live video feed from the selectedtelepresence device and initiate a communication session with thetelepresence device to allow the user of the RPI on PED 3405 to controlthe telepresence device and/or join in a multi-user experience with thetelepresence device. The selection of a patient via work space window3430 may automatically select an associated telepresence device based onavailability, proximity, and/or other preferences. Alternatively, theuser of the RPI on the PED 3405 may additionally select a telepresencedevice. The selection of a patient via work space window 3430 may alsodirect a telepresence robot to navigate to the location of the patient.

FIG. 35 illustrates an embodiment 3500 of an RPI on a PED 3505 includinga touch pad control pane 3540 for navigating a telepresence device whiledisplaying a video feed from a telepresence device in an upper panel3510. According to various embodiments, a one finger tap in the upperpanel 3510 may be used to control the direction in which a head portionof a telepresence device is oriented. A single finger press and drag maydraw a click-to-zoom box. Other touch controls, such as pinch to zoom,mouse-based driving, and/or swiping to move the telepresence device mayalso be available in upper panel 3510. In some embodiments, a four waycontroller (illustrated in FIGS. 27B-27H) may be overlaid within thelive video feed in the upper panel 3510. The touch pad control pane 3540may incorporate various touch controls. For example, a user may swipeleft to gain access to local controls/settings of the PED 3505, swiperight to access controls/settings from the telepresence device, swipedown to access a toolbar, and use multiple fingers to drive thetelepresence device, such as by defining a vector with a magnitude anddirection.

FIG. 36 illustrates an embodiment 3600 of an RPI on a PED 3605,including an avatar display of telepresence devices in a lower panel3620 and a video feed from a telepresence device in an upper panel 3610.In various embodiments, tapping a camera icon may capture an image. Theimage may then appear within a media manager in the lower panel 3620.Drag and drop availability may be available in order for the user tocustomize the telepresence device, such as by adding avatars or specialeffects to the video feed and/or images.

FIG. 37 illustrates an embodiment 3700 of an RPI on a PED 3705 includinga visualization of a telepresence device overlaid on a video feed from atelepresence device in an upper panel 3710. As illustrated, the user mayplace the chess-piece telepresence devices within the video feed in theupper panel 3710 in order to control movement of the actual telepresencedevice. A lower panel 3730 may allow a user to provide othernavigational inputs as well. A lower toolbar 3720 may provide access tovarious settings and/or function to the RPI.

FIG. 38 illustrates another embodiment 3800 of an RPI on a PED 3805including a live video feed in an upper panel 3815, a toolbar beltincluding various media management icons 3817, and settings managertoolbars 3850 and 3860 for the PED 3805 and a telepresence device,respectively.

FIG. 39 illustrates an embodiment 3900 of an RPI on a PED 3905 thatincludes a landing strip navigational panel 3920. A user may provide anintended navigation path that can be displayed on the upper panel 3910and/or the lower panel 3920. The landing strip navigational panel 3920may be used to display an intended navigational path, a directednavigational path (i.e. a path provided by a user to direct thetelepresence device), or a current navigation path on the upper panel3910 as a vector and/or within the lower panel 3920 as a vector and/oras an avatar or rendered image of the telepresence device.

FIG. 40 illustrates an embodiment of an RPI on a PED 4000 including alandscape orientation of a full-screen video feed 4030 from atelepresence device. In addition, a toolbar may be overlaid and/orincluded in a separate lower panel 4020.

FIG. 41 illustrates an embodiment 4100 of an RPI on a PED 4105 includinga joystick-style control 4125 on a touch interface of a lower panel 4120of the PED 4105. The lower panel 4120 may also include information iconsrelating to battery level 4122 and/or network quality 4123. A toolbar4140 may provide access to various settings, features, and/or controls.A full-screen icon 4130 may allow the live video feed in the upper panel4110 to be maximized to a full-screen mode. A window 4115 may beoverlaid in the upper panel 4110 to show the current image captured by acamera 4180 of the PED 4105.

FIG. 42 illustrates an embodiment 4200 of an RPI on a PED 4205 includingdual joystick-style controls 4225 and 4226 on a touch interface of alower panel 4220 of the PED 4205. The lower panel 4220 may also includeinformation icons relating to battery level 4222 and/or network quality4223. A toolbar 4240 may provide access to various settings, features,and/or controls. A full-screen icon 4230 may allow the live video feedin upper panel 4210 to be maximized to a full-screen mode. A window 4215may be overlaid in the upper panel 4210 to show the current imagecaptured by a camera 4280 of the PED 4205.

In the illustrated embodiment, the left joystick 4226 may be configuredto control movement of the base or body of the telepresence device. Theright joystick 4225 may be configured to control movement of a head orupper portion of the telepresence device. The portion of thetelepresence device controlled by each joystick 4225 and 4226 may beuser-selectable and/or reversed.

FIG. 43 illustrates a state diagram 4300 for an RPI for use on a PED. Asillustrated, following startup, a login page 4310 may be displayed.After a username and password are entered, a session notify page 4330may be displayed indicating the network and communication status. Asuccessful login may result in an in-session page being displayed 4332with a live video feed. A dock button may be selected, causing thetelepresence device to display an in-transit page 4325 while itnavigates to a docking station. A stop button may cause the page topause 4327.

From the login page, potentially after successfully logging in, anavigation button may display a navigation page 4320. The navigationpage 4320 may allow a user to select between various navigation modes,such as a map button to result in a map page 4322, or a location buttonto display a list of locations 4324. Again, once a destination isselected, the in-transit page 4325 may be displayed as the telepresencedevice navigates to the selected location.

A settings page 4340 may be displayed allowing a user to select from anynumber of settings. For example, a WiFi selection may result in a WiFipage 4342 being displayed. A robot selection may result in a robot page4344 being displayed. The state diagram 4300 illustrated in FIG. 43 is asimplified state diagram and intentionally omits numerous possiblestates and connections between states for clarity. Each and every panel,icon, setting, application, option, tab, selection, input, and the likedescribed herein may be represented as a separate state, entry action,transition condition, transition, exit action, and/or other component ofa complex state diagram. As will be appreciated by one of skill in theart, each of the various aspects, functions, operations, control panels,icons, objects, buttons, display panels, display windows, etc.,described herein may be described and/or implemented in terms ofsoftware, hardware, and/or firmware and could potentially be describedas a complex state machine.

FIG. 44 illustrates an embodiment of an RPI on a PED 4400 including afull-screen video feed 4410 from a telepresence device. A toolbar 4450and a navigational joystick 4430 are overlaid on the full-screen videofeed 4410. A user may touch, click, or otherwise manipulate the joystickin order to navigate the telepresence device. A dual joystick overlaymay be employed to provide navigational control over a head portion of atelepresence device. In some embodiments, clicking a location within thelive video feed 4410 may control the head movement of the telepresencedevice while the joystick 4430 is intended to control the body movementof the telepresence device. Similarly, a mouse-based driving interface(illustrated in FIGS. 27B-27H) may be overlaid on the full-screen videofeed 4410 instead of or in addition to the joystick 4430.

FIG. 45 illustrates an exemplary toolbar 4500 of icons that may beoverlaid within a page of an RPI and/or inserted as a separate panelwithin a page of an RPI. The icons within toolbar 4500 may provideaccess to charts, vitals, telemetry data, images, lab results, a homepage of the RPI, documents, notes, associated healthcare practitioners,navigation options and features, multimedia control panels, and thelike. Each page with the RPI may include context-based toolbars and/orgeneral toolbars. The toolbar 4500 may be positioned at a bottom, top,edge, and/or other location with respect to the RPI or a pane or panelwithin the RPI. In some embodiments, the toolbar 4500 may be configuredto vanish and selectively reappear, such as, for example, upon amouseover, swipe, or other action.

FIG. 46 illustrates an embodiment 4600 of an RPI on a PED 4605 includingan overlaid instructional panel 4630 describing how a user may manuallydrive a telepresence device. As in previous embodiments, an upper panel4610 may include a live video feed from the telepresence device. Atoolbar 4650 may provide access to various settings, controls, and/orfunctions. For example, the toolbar 4650 may provide access to aheadset, stethoscope, camera, video, navigation, local camera, point,laser, mute, settings for local and remote devices, and/or a disconnect(end) button.

The instructional panel 4630 may provide instructions for finger swipesto move the telepresence device forward and reverse 4631, slide thetelepresence device side to side 4632, rotate the telepresence device tothe right 4633, and rotate the telepresence device to the left 4634.Separate control of a head portion of the telepresence device may beavailable using multiple fingers, using a toggle button (software orhardware), or by tapping within the live video feed 4610. A lowertoolbar 4640 may provide access to various other functions, features,and/or settings of the RPI, such as those described herein andespecially in conjunction with FIG. 45. In some embodiments, the RPI mayinclude instructional or demonstrational videos for any of the variousembodiments or functions described herein.

FIG. 47 illustrates an embodiment 4700 of an RPI on a PED 4705 during amulti-participant telepresence session. As illustrated, a live videofeed of a patient may be displayed in an upper panel 4710 of the PED4705. A toolbar 4750 (and/or 4740) may provide access to various relatedfunctions, settings, and/or controls. A lower panel 4730 may includevideo feeds 4731, 4732, and 4733 from each of three participants in themulti-participant telepresence session. Each of the three participantsmay be using an RPI on a PED and see a similar image of the video feedof the patient in the upper panel 4710. Any number of participants mayparticipate. In some embodiments, each participant may be able tocontrol the telepresence device. In other embodiments, only one, or onlya select few, of the participants may have control of the telepresencedevice.

FIG. 48 illustrates a window or panel 4800 accessible via an RPI on aPED providing access to a care team of a particular patient. The careteam panel 4800 may be presented in a full-screen mode and/or as a panelwithin a display of multiple panels, such as illustrated in FIG. 17. Thecare team panel 4800 may identify the relevant patient by name 4805and/or an identification number 4807. The care team panel 4800 mayinclude a column of healthcare practitioners 4810 associated with thepatient 4805. A column of healthcare practitioner data 4820 may describeeach of the healthcare practitioners 4810 associated with the patient4805. Whether each healthcare practitioner 4810 is on duty or off dutymay be displayed in a third column 4830. A user may also select iconsvia the RPI to consult 4840, text (e.g., an SMS or email) 4850, and/orcall 4860 the associated healthcare practitioner 4810. In variousembodiments, the RPI interface may utilize inputs provided via panel4800 to perform one or more functions via the telepresence device and/ora related telepresence system.

FIG. 49 illustrates an exemplary overlay help screen 4900 accessiblewithin the RPI on a PED to provide instructions regarding availablefunctions on any given screen. In the illustrated example, a user mayhave selected a “help” icon or be in a training mode in order to bepresented with instruction on how to use a particular interface ortoolbar within a screen of the RPI. In the illustrated embodiment, awindow 4950 may display what is currently being captured by a camera ofthe PED and displayed remotely on a display interface of a telepresencedevice. A full-screen image of a live video feed 4910 from atelepresence device may be displayed. In addition, a toolbar 4920 may bedisplayed on a top edge (or other location) of the full-screen displayof the live video feed 4910. The toolbar may be pulled down from ahidden state and/or vanish and reappear, as described herein.

During training, initial use, or after indicating help is needed, thelive video feed 4910 and/or the window 4950 may be dimmed and/orotherwise less obtrusive. A help screen overlay may provide instructionsand/or guidance with how-to toolbars and/or other interface options. Asillustrated in FIG. 49, the overlay may comprise text descriptions ofthe toolbar icons connected by lines to each icon. In some embodiments,instructions and/or guidance for camera controls and/or driving controlsfor the telepresence device may be illustrated as an overlay as well.For example, instructions on how to use the joysticks 4225 and 4226 inFIG. 42 may be provided in a help screen overlay. Similarly, thegestures 4631, 4632, 4633, and 4634 in FIG. 46 may be provided in aninstructional or help screen overlay. In some embodiments, theinstructional overlay may be in the form of moving or video instructionsoverlaid on an existing display. For example, the overlay maydemonstrate the gestures 4631, 4632, 4633, and 4634 in FIG. 46.

According to various embodiments, an RPI may be configured with all orsome of the features and embodiments described herein. For example, anRPI may include any number of the features and embodiments describedherein as selectively displayed and/or selectively functional options.An explicit enumeration of all possible permutations of the variousembodiments is not included herein; however, it will be apparent to oneof skill in the art that any of the variously described embodiments maybe selectively utilized, if not at the same time, in a single RPI.

This disclosure has been made with reference to various exemplaryembodiments, including the best mode. However, those skilled in the artwill recognize that changes and modifications may be made to theexemplary embodiments without departing from the scope of the presentdisclosure. While the principles of this disclosure have been shown invarious embodiments, many modifications may be made to adapt the RPI fora specific environment and/or to satisfy particular operatingrequirements without departing from the principles and scope of thisdisclosure. These and other changes or modifications are intended to beincluded within the scope of the present disclosure. This disclosureincludes all possible permutations of the independent claims and theirdependent claims.

What is claimed is:
 1. A non-transitory computer-readable storage mediumstoring instructions that, when executed by a processor, are configuredto cause the processor to perform operations comprising: retrieving alist of patients; selectively displaying the list of patients; receivinga selection of a patient from the patient list; retrieving an endpointlist including a plurality of remote presence devices and one or moremedical professionals currently associated with particular patients orscheduled to be associated with particular patients for a period of timeto which the respective remote presence devices can navigate; displayingthe endpoint list; receiving a filtering parameter including a name of amedical professional; filtering the endpoint list to display onlyendpoints associated with the name of the medical professional in thefiltering parameter; receiving a selection of a remote presence devicefrom the filtered endpoint list; communicatively connecting anelectronic device to the selected remote presence device; displaying alive video feed from a camera of the selected remote presence device ina video panel on an electronic display of the electronic device; andupdating the live video feed as the selected remote presence devicemoves to the selected patient.
 2. The non-transitory computer-readablestorage medium of claim 1, wherein the operation of communicativelyconnecting an electronic device to a remote presence device isresponsive to receiving a selection of a remote presence device from theendpoint list.
 3. The non-transitory computer-readable storage medium ofclaim 1, wherein the filtering parameter includes at least one ofendpoint type, manufacturer, status, facility, building, floor, room,customer, and group.
 4. The non-transitory computer-readable storagemedium of claim 1, wherein the endpoint list includes a name of aperson.
 5. The non-transitory computer-readable storage medium of claim1, wherein the endpoint list includes at least one of a room name or aroom number.
 6. The non-transitory computer-readable storage medium ofclaim 1, wherein the endpoint list includes a name of a person that isscheduled for a period of time to be associated with the selectedpatient.
 7. The non-transitory computer-readable storage medium of claim1, wherein the operations further comprise: selectively displaying atoolbar comprising at least one selectable setting function associatedwith at least one of the electronic device and the remote presencedevice.
 8. The non-transitory computer-readable storage medium of claim1, wherein the operations further comprise: selectively displaying aplurality of destination locations within a healthcare facility;receiving a selection of one of the plurality of destination locationswithin the healthcare facility as a navigation input; and transmittingnavigation instructions associated with the selected destinationlocation to the remote presence device.
 9. The non-transitorycomputer-readable storage medium of claim 1, wherein the operationsfurther comprise: selectively displaying a map of a healthcare facilityassociated with the remote presence device, and wherein the navigationinput is provided with respect to the map.
 10. The non-transitorycomputer-readable storage medium of claim 1, wherein the operationsfurther comprise: receiving a selection of an avatar to visuallyrepresent an operator of the electronic device; and transmitting theselected avatar to the remote presence device.